MONITOR AND EVALUATE THE ACTION

The Frameworks for Australian Social Statistics can be used to assist in monitoring and evaluating an action put in place in response to an identified social issue. This is the final of three sections describing how the Frameworks for Australian Social Statistics can be used in informing actions and policy:

• Understand and describe the issue
• Formulate actions to support wellbeing
• Monitor and evaluate the action (this section)

Once an action has been implemented, monitoring and evaluation is needed to determine whether it has been successful in achieving the intended aspirations/objectives and outcomes. It is also important to determine whether outputs are effectively reaching those people for whom they are intended.

The following elements of the frameworks should be used to monitor and evaluate and action:

• Actions supporting wellbeing (Responses)
• Our wellbeing (State)

The following key steps should be considered when monitoring and evaluating the action in response to an issue:

• Monitor and evaluate the progress of the action
• Selection of indicators
• Review the selected indicators (if required)

MONITOR AND EVALUATE THE PROGRESS OF THE ACTION

Analysis of data in relation to each of the elements in our wellbeing (State) will determine whether the action is meeting its desired objective or not performing as expected. Using longitudinal analytical techniques or controlled case studies will help to determine the longer term impacts which occur as a result of the action. Performance measurement can assist in determining whether modification of the action is necessary. Measuring impacts tracks the direct effect and performance of the action. Performance measurement showing positive changes to outcomes or impacts may indicate that the action is performing well.

Performance measurement is integral to improving services provided by governments and other providers. What is measured reflects what is important to the community at large, to governments, to service providers, and to the funders of the services (including taxpayers), as well as to consumers and other stakeholders. Benefits of performance measurement include:

• Improved accountability and transparency of service provision to the public, which can be used to create incentives for improved service delivery
• Better information on the effectiveness of changes to policies, practices or programs, as the same measures are reported over time
• Providing the community with an understanding of the availability, quality and effectiveness of services in a particular sector.

See the Australian Institute of Health and Welfare (AIHW), Australia's welfare 2015, Indicators of Australia's welfare.

SELECTION OF INDICATORS

Information provided through the use of good, evidenced based indicators can assist in identifying short term issues that could be rectified and to provide guidance for future actions. Good indicators supported by timely data of good quality allow monitoring of current trends and to evaluate actions. Indicators focused on outcomes measure progress by monitoring key social trends and conditions. Other indicators may focus on monitoring and evaluating process or structural capacity.

It is good to take a flexible approach in selecting indicators in order to maximise the accessibility of information. The selection of indicators can be guided by six principles:

• The indicator used should be the best relevant available indicator and easily understood (interpretability). It is important that indicators are simple to understand and interpret and that the indicator can be easily linked to an objective, decision or activity.
• Indicators should be summary in nature. However, indicators do not need to be ‘all-encompassing’. Indicators may provide useful and meaningful partial or indirect insights, particularly when related to other indicators.
• Indicators do not need to have an extensive time series for inclusion, at least to begin with. However, it is important that a meaningful time series can be created over time. To develop a time series the indicator needs to be reproducible and coherent, sensitive to real world changes and demonstrate clear positive or negative movement over time. The use of standard concepts, classifications, defined population scope and established common methods will assist with this. For more guidance see the Statistical Framework.
• Indicators should be of high or acceptable quality, with specific regard to accuracy, timeliness, and defensible methodology (coherence).
• Indicators should allow disaggregation to focus on key population groups and, if required, comparisons between groups to be made.
• Consider practical issues and requirements including costs of data collection and analysis, impacts and burden on respondents, how the indicator will be presented in reports, timeliness for reporting, and any particular reporting obligations or legislative requirements.

For examples of how to select appropriate indicators see ABS Measures of Australia's Progress, 2013. This publication describes the criteria used to select and assess the quality of a wide range of national progress indicators.

For further guidance on how to assess the quality of indicators in terms of relevance, timeliness, accuracy, coherence, interpretability, accessibility, see ABS Data Quality Framework, May 2009.

When selecting indicators consider each element of the conceptual framework to determine what information will be needed to create a full picture of the issue and its impacts.

REVIEW THE SELECTED INDICATORS IF REQUIRED

Different indicators may be required if the selected indicators are not monitoring and evaluating the impact of the action effectively. This step may determine that there is a gap in the statistical information available. Some statistical measurement issues which may impact on the quality and availability of data are discussed in the statistical framework.

Determine that the indicators:

• Are still relevant, easy to understand and can be linked to the action
• Provide timely information and can form a time series
• Allow focus to be on the population group of interest and can be compared with other groups
• Ensure the collection of data is not costly or burdensome to respondents

The following table can be used to assist with monitoring and evaluating an action:

Step	Key Questions
Monitor and evaluate the progress of the action	Does the assessment against the benchmarks show that the action is meeting initial objectives? Does the evaluation provide insight and identification of social areas that require improvement? Were there issues relating to the appropriateness of the action to the particular needs of the target population? What actions worked and what should be changed? Did the action create new pressures? What was the overall impact on wellbeing?
Select indicators	Are the selected indicators relevant and easily understood? Are the selected indicators coherent and of a high quality? Do the selected indicators allow for desired disaggregations and comparisons? Are the selected indicators practical and easy to report on?
Review the selected indicators	Were the chosen indicators and benchmarks appropriate for evaluation and monitoring of the actions?